Skip to content

Draft: Implement watching file paths via oslib.watch (for 0.12.x branch) #5073

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
May 19, 2025

Conversation

arturaz
Copy link
Contributor

@arturaz arturaz commented May 6, 2025

Backport of #5068

The fs watching is disabled by default and can be enabled via --notify-watch=true.

@arturaz arturaz changed the base branch from main to 0.12.x May 6, 2025 09:26
@arturaz arturaz force-pushed the improvement/fs-watching-0.12.x branch from 25e1fb5 to caffff7 Compare May 8, 2025 09:40
@arturaz arturaz force-pushed the improvement/fs-watching-0.12.x branch from 7c1395f to 063e47f Compare May 8, 2025 09:53
@arturaz
Copy link
Contributor Author

arturaz commented May 8, 2025

Noticed this popping up while testing (note for self):

Exception in thread "Thread-535" java.nio.channels.ClosedChannelException
        at java.base/sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:160)
        at java.base/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:284)
        at java.base/sun.nio.ch.ChannelOutputStream.writeFully(ChannelOutputStream.java:68)
        at java.base/sun.nio.ch.ChannelOutputStream.write(ChannelOutputStream.java:105)
        at java.base/java.io.OutputStream.write(OutputStream.java:124)
        at mill.runner.Watching$.writeToWatchLog$1(Watching.scala:120)
        at mill.runner.Watching$.$anonfun$watchAndWait$15(Watching.scala:184)
        at mill.runner.Watching$.$anonfun$watchAndWait$15$adapted(Watching.scala:183)
        at os.watch.WatchServiceWatcher.run(WatchServiceWatcher.scala:152)
        at os.watch.package$$anon$1.run(package.scala:45)

@lihaoyi lihaoyi marked this pull request as draft May 13, 2025 05:26
@lihaoyi lihaoyi marked this pull request as ready for review May 19, 2025 08:35
lihaoyi pushed a commit that referenced this pull request May 19, 2025
Currently watching files uses polling which is inefficient with large
codebases. On my laptop `./mill --watch` uses 20% CPU just to watch the
files for changes.

This uses `oslib`'s `watch` module to watch for the file changes instead
of polling them.

We had to update oslib (com-lihaoyi/os-lib#386)
to:
- add a capability of filtering what folders it watches. This is needed
because watching large generated folders like `out` or `.bloop` emits
Java FS watcher `OVERFLOW` events.
- remove the random `println`'s that `oslib.watch` had.

The current approach to watching is to watch the workspace root via FS
watching. FS watching only works with folders and we have to watch
`root/build.mill`, and thus, `root/`, so everything else falls under it.

Mac OS watcher performs watching recursively natively, but on linux
oslib adds recursive watches itself, so we use `oslib.watch` filter to
prevent watching anything that is not an ancestor of watched root. For
example, if we have source at `root/module-a/src/`, we'll watch `root/`,
`root/module-a/` and `root/`.

Finally, events are filtered so that unrelated changes (like creating
`root/random-file.txt`) does not trigger reevaluation.

The fs watching is enabled by default and can be disabled via
`--watch-via-fs-notify=false`.

`0.12.x` version: #5073

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
@lihaoyi lihaoyi merged commit 5dd2bd3 into com-lihaoyi:0.12.x May 19, 2025
31 of 33 checks passed
@lefou lefou added this to the 0.12.14 milestone May 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants